Response to Comment on "Pushing the frontiers of density functionals by solving the fractional electron problem".

Gerasimov et al. claim that the ability of DM21 to respect fractional charge (FC) and fractional spin (FS) conditions outside of the training set has not been demonstrated in our paper. This is based on (i) asserting that the training set has a ~50% overlap with our bond-breaking benchmark (BBB) and (ii) questioning the validity and accuracy of our other generalization examples. We disagree with their analysis and believe that the points raised are either incorrect or not relevant to the main conclusions of the paper and to the assessment of general quality of DM21.

Pushing the frontiers of density functionals by solving the fractional electron problem.

Density functional theory describes matter at the quantum level, but all popular approximations suffer from systematic errors that arise from the violation of mathematical properties of the exact functional. We overcame this fundamental limitation by training a neural network on molecular data and on fictitious systems with fractional charge and spin. The resulting functional, DM21 (DeepMind 21), correctly describes typical examples of artificial charge delocalization and strong correlation and performs better than traditional functionals on thorough benchmarks for main-group atoms and molecules. DM21 accurately models complex systems such as hydrogen chains, charged DNA base pairs, and diradical transition states. More crucially for the field, because our methodology relies on data and constraints, which are continually improving, it represents a viable pathway toward the exact universal functional.

Exact Density Functional Obtained via the Levy Constrained Search.

A stochastic minimization method for a real-space wave function, Ψ(r1, r2...r n), constrained to a chosen density, ρ(r), is developed. It enables the explicit calculation of the Levy constrained search, F[ρ] = minΨ→ρ⟨Ψ| T̂ + V̂ ee|Ψ⟩, which gives the exact functional of density functional theory. This general method is illustrated in the evaluation of F[ρ] for densities in one dimension with a soft-Coulomb interaction. Additionally, procedures are given to determine the first and second functional derivatives, δ F/δρ(r) and δ2 F/[δρ(r)δρ(r')]. For a chosen external potential, v(r), the functional and its derivatives are used in minimizations over densities to give the exact energy, E v, without needing to solve the Schrödinger equation.

Fermionic Statistics in the Strongly Correlated Limit of Density Functional Theory.

Exact pieces of information on the adiabatic connection integrand, Wλ[ρ], which allows evaluation of the exchange-correlation energy of Kohn-Sham density functional theory, can be extracted from the leading terms in the strong coupling limit (λ → ∞, where λ is the strength of the electron-electron interaction). In this work, we first compare the theoretical prediction for the two leading terms in the strong coupling limit with data obtained via numerical implementation of the exact Levy functional in the simple case of two electrons confined in one dimension, confirming the asymptotic exactness of these two terms. We then carry out a first study on the incorporation of the Fermionic statistics at large coupling λ, both numerical and theoretical, confirming that spin effects enter at orders ∼e-√λ.

Local scaling correction for reducing delocalization error in density functional approximations.

Delocalization error is one of the most fundamental and dominant errors that plagues presently used density functional approximations. It is responsible for a large class of problems in the density functional theory calculations. For an effective and universal alleviation of the delocalization error, we develop a local scaling correction scheme by imposing the Perdew-Parr-Levy- Balduz linearity condition to local regions of a system. Our novel scheme is applicable to various mainstream density functional approximations. It substantially reduces the delocalization error, as exemplified by the significantly improved description of dissociating molecules, transition-state species, and charge-transfer systems. The usefulness of our novel scheme affirms that the explicit treatment of fractional electron distributions is essentially important for reducing the intrinsic delocalization error associated with approximate density functionals.

Qualitative breakdown of the unrestricted Hartree-Fock energy.

The stretching of closed-shell molecules is a qualitative problem for restricted Hartree-Fock that is usually circumvented by the use of unrestricted Hartree-Fock (UHF). UHF is well known to break the spin symmetry at the Coulson-Fischer point, leading to a discontinuous derivative in the potential energy surface and incorrect spin density. However, this is generally not considered as a major drawback. In this work, we present a set of two electron molecules which magnify the problem of symmetry breaking and lead to drastically incorrect potential energy surfaces with UHF. These molecules also fail with unrestricted density-functional calculations where a functional such as B3LYP gives both symmetry breaking and an unphysically low energy due to the delocalization error. The implications for density functional theory are also discussed.

The derivative discontinuity of the exchange-correlation functional.

The derivative discontinuity is a key concept in electronic structure theory in general and density functional theory in particular. The electronic energy of a quantum system exhibits derivative discontinuities with respect to different degrees of freedom that are a consequence of the integer nature of electrons. The classical understanding refers to the derivative discontinuity of the total energy as a function of the total number of electrons (N), but it can also manifest at constant N. Examples are shown in models including several hydrogen systems with varying numbers of electrons or nuclear charge (Z), as well as the 1-dimensional Hubbard model (1DHM). Two sides of the problem are investigated: first, the failure of currently used approximate exchange-correlation functionals in DFT and, second, the importance of the derivative discontinuity in the exact electronic structure of molecules, as revealed by full configuration interaction (FCI). Currently, all approximate functionals, including hybrids, miss the derivative discontinuity, leading to basic errors that can be seen in many ways: from the complete failure to give the total energy of H2 and H2(+), to the missing gap in Mott insulators such as stretched H2 and the thermodynamic limit of the 1DHM, or a qualitatively incorrect density in the HZ molecule with two electrons and incorrect electron transfer processes. Description of the exact particle behaviour of electrons is emphasised, which is key to many important physical processes in real systems, especially those involving electron transfer, and offers a challenge for the development of new exchange-correlation functionals.

Fractional charge behavior and band gap predictions with the XYG3 type of doubly hybrid density functionals.

In this work, we examine the fractional charge behaviors of doubly hybrid (DH) functionals. By plotting the ground-state energies E and energy derivatives for atoms and molecules with fractional electron numbers N, we directly quantify the delocalization errors of some representative DH functionals such as B2PLYP, XYG3, and XYGJ-OS. Numerical assessments on ionization potentials (IPs), electron affinities (EAs), and fundamental gaps, from either integer number calculations or energy derivative calculations, are provided. It is shown that the XYG3 type of DH functionals gives good agreement between their energy derivatives and the experimental IPs, EAs, and gaps, as expected from their nearly straight line fractional charge behaviors.

Dramatic changes in electronic structure revealed by fractionally charged nuclei.

Discontinuous changes in the electronic structure upon infinitesimal changes to the Hamiltonian are demonstrated. These are revealed in one and two electron molecular systems by full configuration interaction (FCI) calculations when the realm of the nuclear charge is extended to be fractional. FCI electron densities in these systems show dramatic changes in real space and illustrate the transfer, hopping, and removal of electrons. This is due to the particle nature of electrons seen in stretched systems and is a manifestation of an energy derivative discontinuity at constant number of electrons. Dramatic errors of density functional theory densities are seen in real space as this physics is missing from currently used approximations. The movements of electrons in these simple systems encapsulate those in real physical processes, from chemical reactions to electron transport and pose a great challenge for the development of new electronic structure methods.

Extension of many-body theory and approximate density functionals to fractional charges and fractional spins.

The exact conditions for density functionals and density matrix functionals in terms of fractional charges and fractional spins are known, and their violation in commonly used functionals has been shown to be the root of many major failures in practical applications. However, approximate functionals are designed for physical systems with integer charges and spins, not in terms of the fractional variables. Here we develop a general framework for extending approximate density functionals and many-electron theory to fractional-charge and fractional-spin systems. Our development allows for the fractional extension of any approximate theory that is a functional of G(0), the one-electron Green's function of the non-interacting reference system. The extension to fractional charge and fractional spin systems is based on the ensemble average of the basic variable, G(0). We demonstrate the fractional extension for the following theories: (1) any explicit functional of the one-electron density, such as the local density approximation and generalized gradient approximations; (2) any explicit functional of the one-electron density matrix of the non-interacting reference system, such as the exact exchange functional (or Hartree-Fock theory) and hybrid functionals; (3) many-body perturbation theory; and (4) random-phase approximations. A general rule for such an extension has also been derived through scaling the orbitals and should be useful for functionals where the link to the Green's function is not obvious. The development thus enables the examination of approximate theories against known exact conditions on the fractional variables and the analysis of their failures in chemical and physical applications in terms of violations of exact conditions of the energy functionals. The present work should facilitate the calculation of chemical potentials and fundamental bandgaps with approximate functionals and many-electron theories through the energy derivatives with respect to the fractional charge. It should play an important role in developing accurate approximate density functionals and many-body theory.

Derivative discontinuity, bandgap and lowest unoccupied molecular orbital in density functional theory.

The conventional analysis of Perdew and Levy, and Sham and Schlüter shows that the functional derivative discontinuity of the exchange-correlation density functional plays a critical role in the correct prediction of bandgaps, or the chemical hardness. In a recent work by the present authors, explicit expressions for bandgap prediction with all common types of exchange-correlation functionals have been derived without invoking the concept of exchange-correlation energy functional derivative discontinuity at all. We here analyze the two approaches and establish their connection and difference. The present analysis further leads to several important results: (1) The lowest unoccupied molecular orbital (LUMO) in DFT has as much meaning in describing electron addition as the highest occupied molecular orbital (HOMO) in describing electron removal. (2) Every term in the total energy functional contributes to the energy gap because of the discontinuity of the derivative of the density (or density matrix) with respect to the number of electrons, ((∂ρ(s)(r('),r))/∂N)(v(s) ), at integers. (3) Consistent with the Perdew-Levy-Sham-Schlüter conclusion that the exact Kohn-Sham energy gap differs from the fundamental bandgap by a finite correction due to the functional derivative discontinuity of the exchange-correlation energy, we show that the exchange-correlation functional cannot be an explicit and differentiable functional of the electron density, either local or nonlocal. The last result is further strengthened when we consider Mott insulators. There, the exact exchange-correlation functional needs to have an explicitly discontinuous (nondifferentiable) dependence on the density or the density matrix. (4) We obtain exact conditions on the derivatives of total energy with respect to the spin-up and spin-down number of electrons.

Improving band gap prediction in density functional theory from molecules to solids.

A novel nonempirical scaling correction method is developed to tackle the challenge of band gap prediction in density functional theory. For finite systems the scaling correction largely restores the straight-line behavior of electronic energy at fractional electron numbers. The scaling correction can be generally applied to a variety of mainstream density functional approximations, leading to significant improvement in the band gap prediction. In particular, the scaled version of a modified local density approximation predicts band gaps with an accuracy consistent for systems of all sizes, ranging from atoms and molecules to solids. The scaled modified local density approximation thus provides a useful tool to quantitatively characterize the size-dependent effect on the energy gaps of nanostructures.

Revealing noncovalent interactions.

Molecular structure does not easily identify the intricate noncovalent interactions that govern many areas of biology and chemistry, including design of new materials and drugs. We develop an approach to detect noncovalent interactions in real space, based on the electron density and its derivatives. Our approach reveals the underlying chemistry that compliments the covalent structure. It provides a rich representation of van der Waals interactions, hydrogen bonds, and steric repulsion in small molecules, molecular complexes, and solids. Most importantly, the method, requiring only knowledge of the atomic coordinates, is efficient and applicable to large systems, such as proteins or DNA. Across these applications, a view of nonbonded interactions emerges as continuous surfaces rather than close contacts between atom pairs, offering rich insight into the design of new and improved ligands.

Discontinuous nature of the exchange-correlation functional in strongly correlated systems.

Standard approximations for the exchange-correlation functional have been found to give big errors for the linearity condition of fractional charges, leading to delocalization error, and the constancy condition of fractional spins, leading to static correlation error. These two conditions are now unified and extended to states with both fractional charge and fractional spin to give a much more stringent condition: the exact energy functional is a plane, linear along the fractional charge coordinate and constant along the fractional spin coordinate with a line of discontinuity at the integer. Violation of this condition underlies the failure of all known approximate functionals to describe the gaps in strongly correlated systems. It is shown that explicitly discontinuous functionals of the density or orbitals that go beyond these currently used smooth approximations is the key for the application of density functional theory to strongly correlated systems.

Second-Order Perturbation Theory with Fractional Charges and Fractional Spins.

In this work the behavior of MP2 for fractional occupations is investigated. The consideration of fractional charge behavior gives a simple derivation of an expression for the chemical potential (or the derivative of energy with respect to the number of electrons) of MP2. A generalized optimized effective potential formalism (OEP) has been developed in which the OEP is a nonlocal potential, which can be applied to explicit functionals of the orbitals and eigenvalues and also facilitates the evaluation of the chemical potential. The MP2 derivative improves upon the corresponding Koopmans' theorem in Hartree-Fock theory for the ionization energy and also gives a good estimate of the electron affinity. In strongly correlated systems with degeneracies and fractional spins, MP2 diverges, and another corrected second-order perturbative method ameliorates this failure for the energy but still does not recapture the correct behavior for the energy derivatives that yield the gap. Overall we present a view of wave function based methods and their behavior for fractional charges and spins that offers insight into the application of these methods to challenging chemical problems.

Fractional spins and static correlation error in density functional theory.

Electronic states with fractional spins arise in systems with large static correlation (strongly correlated systems). Such fractional-spin states are shown to be ensembles of degenerate ground states with normal spins. It is proven here that the energy of the exact functional for fractional-spin states is a constant, equal to the energy of the comprising degenerate pure-spin states. Dramatic deviations from this exact constancy condition exist with all approximate functionals, leading to large static correlation errors for strongly correlated systems, such as chemical bond dissociation and band structure of Mott insulators. This is demonstrated with numerical calculations for several molecular systems. Approximating the constancy behavior for fractional spins should be a major aim in functional constructions and should open the frontier for density functional theory to describe strongly correlated systems. The key results are also shown to apply in reduced density-matrix functional theory.

Delocalization errors in density functionals and implications for main-group thermochemistry.

The difficulty of approximate density functionals in describing the energetics of Diels-Alder reactions and dimerization of aluminum complexes is analyzed. Both of these reaction classes involve formation of cyclic or bicyclic products, which are found to be underbound by the majority of functionals considered. We present a consistent view of these results from the perspective of delocalization error. This error causes approximate functionals to give too low energy for delocalized densities or too high energy for localized densities, as in the cyclic and bicyclic reaction products. This interpretation allows us to understand better a wide range of errors in main-group thermochemistry obtained with popular density functionals. In general, functionals with minimal delocalization error should be used for theoretical studies of reactions where there is a loss of extended conjugation or formation of highly branched, cyclic, and cagelike molecules.

Insights into current limitations of density functional theory.

Density functional theory of electronic structure is widely and successfully applied in simulations throughout engineering and sciences. However, for many predicted properties, there are spectacular failures that can be traced to the delocalization error and static correlation error of commonly used approximations. These errors can be characterized and understood through the perspective of fractional charges and fractional spins introduced recently. Reducing these errors will open new frontiers for applications of density functional theory.

Localization and delocalization errors in density functional theory and implications for band-gap prediction.

The band-gap problem and other systematic failures of approximate exchange-correlation functionals are explained from an analysis of total energy for fractional charges. The deviation from the correct intrinsic linear behavior in finite systems leads to delocalization and localization errors in large and bulk systems. Functionals whose energy is convex for fractional charges such as the local density approximation display an incorrect apparent linearity in the bulk limit, due to the delocalization error. Concave functionals also have an incorrect apparent linearity in the bulk calculation, due to the localization error and imposed symmetry. This resolves an apparent paradox and identifies the physical nature of the error to be addressed to obtain accurate band gaps from density functional theory.